Skip to content

fix(interruption): keep an open overlap when a second agent speech segment starts - #6550

Closed
biztex wants to merge 1 commit into
livekit:mainfrom
biztex:fix/interruption-second-segment-overlap
Closed

fix(interruption): keep an open overlap when a second agent speech segment starts#6550
biztex wants to merge 1 commit into
livekit:mainfrom
biztex:fix/interruption-second-segment-overlap

Conversation

@biztex

@biztex biztex commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Fixes #6548

Problem

One agent turn can span several speech segments: a reply that produces tool calls deliberately skips _on_end_of_agent_speech, so the follow-up tool-response reply (or a queued say()) raises _AgentSpeechStartedSentinel a second time with no end sentinel in between. _forward_data treated every start sentinel as a brand-new turn and called _reset_state(), clearing _overlap_started while the user was mid-interruption.

From that moment the interruption is silently swallowed:

  • _overlap_started gates whether user audio is forwarded to the bargein gateway at all, so classification stops;
  • the flag can only be raised again by _OverlapSpeechStartedSentinel, which only comes from a fresh VAD speech onset — and VAD does not re-announce speech that is already under way;
  • at _OverlapSpeechEndedSentinel the event emission is behind if self._overlap_started, so no OverlappingSpeechEvent is emitted and AgentActivity.on_interruption never runs. The agent talks straight through the user.

Fix

Skip the reset when the stream already believes the agent is speaking and an overlap is open — i.e. treat the sentinel as a continuation of the same turn. The guard is deliberately narrow: a genuine new turn (an end sentinel was seen, so _agent_speech_started is False) still wipes the overlap, the audio buffer, the cache and the counters.

This ports the stream-side guard from the JS fix (livekit/agents-js#2117). The other half of that PR — restoring overlap state on a transport-failover replacement stream — does not apply to Python: these flags live on self and _run() reconnects in place on the same stream object.

Tests

New tests/test_interruption/test_interruption_overlap_state.py (unit, hermetic — fake gateway WS that answers every batch), ported from the JS PR's regression suite:

  • test_second_speech_segment_keeps_open_overlap — user audio keeps reaching the gateway after a mid-overlap second segment, and the end-of-overlap event still surfaces with num_requests > 0 (fails on main)
  • test_bargein_verdict_after_second_speech_segment — a bargein_detected response after the second segment still produces is_interruption=True (fails on main)
  • test_new_agent_turn_still_resets_overlap_state — a genuine new turn still resets: audio stays gated until a fresh overlap opens (passes on main and with the fix — pins the guard's narrowness)

Verification: full --unit suite passes (1274 passed), ruff format --check / ruff check clean, mypy strict clean.

🤖 Generated with Claude Code

@biztex
biztex requested a review from a team as a code owner July 26, 2026 17:19
@CLAassistant

CLAassistant commented Jul 26, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

…gment starts

One agent turn can span several speech segments: a reply that produces tool
calls skips _on_end_of_agent_speech, so the follow-up tool-response reply (or a
queued say()) raises _AgentSpeechStartedSentinel again with no end sentinel in
between. _forward_data treated every start sentinel as a new turn and reset the
stream state, clearing _overlap_started while the user was still mid-interrupt.

From that point the interruption was silently swallowed: _overlap_started gates
audio forwarding to the bargein gateway, and only a fresh VAD speech onset -
which never comes for speech already under way - can raise it again, so no
verdict was ever emitted and the agent talked straight through the user.

Skip the reset when the stream already believes the agent is speaking and an
overlap is open, treating the sentinel as a continuation of the same turn. A
genuine new turn (an end sentinel was seen) still wipes the overlap, the cache
and the counters.

Ports the stream-side guard from livekit/agents-js#2117 (the transport-failover
half of that PR does not apply here: these flags live on self and _run()
reconnects in place on the same stream object).

Fixes livekit#6548
@biztex
biztex force-pushed the fix/interruption-second-segment-overlap branch from ba01151 to 510f401 Compare July 27, 2026 00:14
@biztex

biztex commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

TL;DR for reviewers

One agent turn can span several speech segments (the reply after a tool call, or a queued say()). The second segment's start sentinel used to hit _reset_state() mid-overlap, which silently dropped an in-progress user barge-in: audio stopped reaching the classifier, the flag could never re-arm (VAD does not re-announce ongoing speech), and no verdict was emitted — the agent talked straight through the user.

The change is a narrow continuation guard in _forward_data (mirrors the JS fix in livekit/agents-js#2117), plus three hermetic regression tests: audio keeps flowing across a second segment, a bargein verdict still surfaces afterwards, and a genuine new turn still resets everything.

Status: all checks green, CLA signed, no open review threads. Happy to adjust anything.

@biztex

biztex commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #6564, which fixes the root cause at the signal source (real speech boundaries around the thinking gap, restarting overlap inference on resume) and also covers the backchannel-metrics and remote-session angles. Closing in its favor.

@biztex biztex closed this Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adaptive interruption: a second agent speech segment (e.g. after a tool call) disarms an open overlap and silently swallows the interruption

2 participants